Repository navigation
Write .b2z stores and downloads with the umask's permissions again, not 0600 - #789
Open
FabianIsensee wants to merge 1 commit into
Open
FabianIsensee wants to merge 1 commit into
FabianIsensee wants to merge 1 commit into
Conversation
Since to_b2z() became atomic (3ea9619, 4.8.0) it builds the zip in tempfile.mkstemp() and os.replace()s it onto the target. mkstemp() always creates 0o600, so every .b2z written or rewritten since then is private to its owner whatever the umask, and a file the user had chmod-ed loses its mode. On shared filesystems the group can no longer read the stores. RemoteStore exports, RemoteFile/OrdinaryFile downloads and the RemoteStore disk cache's manifests use the same pattern and have the same problem. Add core.mkstemp_for_replace(): it creates the temporary file beside the target with os.open(..., O_CREAT | O_EXCL, 0o666), so the kernel applies the umask (or the directory's default ACL) exactly as for a plain open(); the umask is never read, since os.umask() can only be read by setting it process-wide, which races with other threads. If the target exists, the temporary file is created no more open than it and then gets its permission bits (fchmod), as an in-place write would have kept them (4.7.0 behaviour). Windows is unchanged. Use it at every mkstemp() + os.replace() site; to_b2z() and the RemoteStore export now write the zip through the fd, so a read-only target can still be replaced, as on main. publish_materialized() keeps a replaced .b2z's mode too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwMZj6jcia1iL27SPfQCoh
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since 4.8.0 every
.b2zthat DictStore/TreeStore write or rewrite ends up with mode 0600, whatever the umask:to_b2z()builds the zip intempfile.mkstemp(), which always creates 0o600, andos.replace()s it onto the target (3ea9619, "Make to_b2z() atomic"). On shared filesystems the group and other users can no longer read the stores, and a store the user had chmod-ed (say 0640) drops to 0600 on its next rewrite. To reproduce:umask 002, thenwith blosc2.TreeStore("t.b2z", mode="w") as t: t["/a"] = np.arange(3)gives-rw-------on 4.8.0, 4.14.1 and main; 4.7.0 gave-rw-rw-r--..b2d,.b2eand.b2ndfiles are not affected.The same
mkstemp()+os.replace()pattern is used byRemoteStore.save()(.b2zexports),RemoteFile.download(), the b2viewOrdinaryFile.download()andremote_store_cache.atomic_write()(RemoteStore disk cache files such asactive_generation.json, which ended up 0600 next to 0644 arrays in the same cache directory), so their files were 0600 too.materialize()to.b2zand RemoteCTable saves to.b2zwere affected throughto_b2z().This PR adds
blosc2.core.mkstemp_for_replace()and uses it at those sites instead ofmkstemp():os.open(..., O_CREAT | O_EXCL, 0o666), so the kernel applies the umask (or the directory's default ACL) exactly as for a plainopen(). The umask itself is never read:os.umask()can only be read by setting it, process-wide, which would race with other threads creating files.target_mode & 0o666) and then set withfchmod()on the fd, so nobody can open it with wider permissions in between. The chmod is best effort: filesystems that refuse it (e.g. vfat) keep their own modes.to_b2z()and the RemoteStore export now write the zip through that fd instead of reopening the temporary file by path, so a read-only.b2zcan still be replaced as on main (and stays read-only).publish_materialized()keeps a replaced.b2z's mode too.Files written 0600 by 4.8.0-4.14.x keep that mode when they are rewritten; a
chmodfixes them once.Results, umask 022 / 002 (main vs this PR; for every
.b2zrow, 4.7.0 gave the same as this PR):.b2z(DictStore/TreeStore close,to_b2z(),materialize(),RemoteStore.save()), downloads, cache files: 0600 / 0600 → 0644 / 0664.b2z(mode "a" or "w",to_b2z(overwrite=True),materialize(overwrite=True),save(overwrite=True)) or a 0640 download target: 0600 → 0640g::rwx,o::---), umask 022:open()gives 0660, main 0600, this PR 0660Tests:
tests/test_file_mode.py(umask 022 and 002; skipped on Windows and WASM) covers:.b2zstores (modes "w" and "a"),to_b2z()andmaterialize();RemoteStore.save()plus its disk cache files;OrdinaryFile.download()andRemoteFile.download();.b2z.15 of the 17 fail on main; the other 2 check the temporary file's mode at creation, which mkstemp's 0600 always passes. Full suite: same results as main apart from the new tests. No performance impact: one
statper write, plus achmodwhen the target exists.I used Claude Code (Anthropic) to find the affected writers, write the fix and tests, and draft this description; I reviewed the change.
🤖 Generated with Claude Code